Scan-based messaging for electronic eyewear devices

ABSTRACT

Systems and methods are described for providing scan based imaging using an electronic eyewear device. The methods include scanning a scene using the electronic eyewear device to capture at least one image in an environment of a first user and identifying at least one physical marker in the scanned scene. Upon identification of the at least one physical marker in the scanned scene, a message (e.g., preselected AR content) is passively sent to at least one of directly to a second user or to at least one physical marker at a remote location for presentation to the second user. The message may be sent without use of the first user&#39;s hands to make a selection. The physical markers may be associated with a fixed object, a human face, a pet, a vehicle, a person, a logo, and the like.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 63/240,122 filed on Sep. 2, 2021, the contents of which areincorporated fully herein by reference.

TECHNICAL FIELD

Examples set forth in the present disclosure relate to systems andmethods for providing messaging from portable electronic devices,including wearable electronic devices such as smart glasses. Moreparticularly, but not by way of limitation, the present disclosuredescribes systems and methods to enable wearers of electronic eyeweardevices to scan a scene and send a message in a passive, hands-freemanner.

BACKGROUND

Wearable electronic devices such as electronic eyewear devices maycommunicate with application programs running on mobile devices such asa user's mobile computing device and, in some cases, may communicatedirectly with a server. In either case, the electronic eyewear devicemay support direct device integration with communication applicationbackend services as well as third-party application programminginterfaces (APIs) such as text-to-speech, the SHAZAM PLAYER® app, objectrecognition, and the like. The wearer of the electronic eyewear devicesmay select display features through interaction with the electroniceyewear device.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the various implementations disclosed will be readilyunderstood from the following detailed description, in which referenceis made to the appended drawing figures. A reference numeral is usedwith each element in the description and throughout the several views ofthe drawing. When a plurality of similar elements is present, a singlereference numeral may be assigned to like elements, with an addedlower-case letter referring to a specific element.

The various elements shown in the figures are not drawn to scale unlessotherwise indicated. The dimensions of the various elements may beenlarged or reduced in the interest of clarity. The several figuresdepict one or more implementations and are presented by way of exampleonly and should not be construed as limiting. Included in the drawingare the following figures:

FIG. 1A illustrates a side view of an example hardware configuration ofan electronic eyewear device showing a right optical assembly with animage display;

FIG. 1B illustrates a top cross-sectional view of a temple of theelectronic eyewear device of FIG. 1A;

FIG. 2A illustrates a rear view of an example hardware configuration ofan electronic eyewear device in an example hardware configuration;

FIG. 2B illustrates a rear view of an example hardware configuration ofanother electronic eyewear device in an example hardware configuration;

FIG. 2C and FIG. 2D illustrate rear views of example hardwareconfigurations of an electronic eyewear device including two differenttypes of image displays;

FIG. 3 illustrates an example of visible light captured by the leftvisible light camera as a left raw image and visible light captured bythe right visible light camera as a right raw image;

FIG. 4 illustrates a block diagram of electronic components of theelectronic eyewear device in a system for providing social connectionsthrough objects in a sample configuration;

FIG. 5 illustrates a block diagram of electronic components of a mobiledevice adapted for use with system of FIG. 4 ;

FIG. 6 illustrates a sample configuration of a computer system adaptedto implement the server of the system of FIG. 4 in a sampleconfiguration;

FIG. 7 illustrates establishing a first object as a marker for a firstuser for establishing a social connection in a sample configuration;

FIG. 8 illustrates completion of a social connection by establishing asecond object as a marker for a second user in a sample configuration;

FIG. 9 illustrates the transmission of sparkles from the first user tothe second user by the first user simply glancing at the first objectestablished as a marker;

FIG. 10 illustrates the reception of sparkles by the second user uponthe second user glancing at the second object established as a marker;

FIG. 11 illustrates the transmission of an object segmented from thesurroundings (a mug) for transmission by the second user to the firstuser via the connection between the first and second objects;

FIG. 12 illustrates receipt of the object (mug) transmitted to the firstuser by the second user via the connection between the first and secondobjects;

FIG. 13 illustrates sample connections of objects for sociallyconnecting the first and second users by sending sparkles to indicatethat the respective users are present, in a sample configuration;

FIG. 14 illustrates sample connections of objects for sociallyconnecting the first and second users by sending objects between theusers, in a sample configuration;

FIG. 15 illustrates a flow chart implemented by the mobile device forproviding object pairing and managing connections in a sampleconfiguration; and

FIG. 16 illustrates a flow chart implemented by the electronic eyeweardevice for identifying and recognizing marker-endpoint objects andprompting the sending of at least one of AR objects or images from theenvironment for socially connecting respective users in a sampleconfiguration.

DETAILED DESCRIPTION

This disclosure is directed to systems and methods for providing scanbased imaging using an electronic eyewear device. The methods includescanning a scene using the electronic eyewear device to capture at leastone image in an environment of a first user and identifying at least onephysical marker in the scanned scene. Upon identification of the atleast one physical marker in the scanned scene, a message (e.g.,preselected AR content) is passively sent to at least one of directly toa second user or to at least one physical marker at a remote locationfor presentation to the second user on the second user's electroniceyewear device. The message may be sent without use of the first user'shands to make a selection. The physical markers may be associated with afixed object, a human face, a pet, a vehicle, a person, a logo, and thelike. The electronic eyewear device is thus adapted to passively sendmessages to physical markers at the remote location for presentation tothe second user as the first user moves through the environment of thefirst user and encounters physical markers previously paired with thephysical markers at the remote location. Thus, when the first user seesa physical marker, pre-configured content in the form of a 3D snapshotof a real object, AR content from an AR gallery, scene-specific ARcontent, and the like may be sent with the message to the second userwithout further actions on the part of the first user. In a sampleconfiguration, the snapshot is a 3D model of the real world object thatis so realistic that the receiver feels as if the sender has sent theobject itself In this example, the 3D model may blend seamlessly withthe physical environment as if it were a real object.

Generally, a flexible, user-customizable platform is described forproviding social connection between users of electronic eyewear devices.The systems and methods described herein enable wearers of electroniceyewear devices to indirectly interact with one another by establishingobjects as personalized anchor points for social connection. The systemsand methods enable a user to feel connected to another user by creatingawareness of what a remote friend (who also has a compatible electroniceyewear device) is doing by creating distributed and connected“wormholes” using objects in the environments of the respective users.The remote friends may stay connected to each other by creatinginterpersonal awareness of what each friend is doing throughout the dayusing the “wormholes” between the connected objects. Various forms ofconfigurable ambient awareness displays are integrated into the physicalenvironment of wearers of the electronic eyewear devices to supportinterpersonal awareness and to foster social and emotional connectionswith other wearers of the electronic eyewear devices.

The systems and methods described herein thus enable socialconnection/presence through distributed and connected real-worldobjects. The connections may be symmetric (between like objects) orasymmetric (between different types of objects), and the content that istransferred may be abstract augmented reality (AR) content or real-worldcontent. The systems and methods also support transient and persistentAR content. The transfer of such content via the distributed andconnected real-world objects enables remote friends to indirectlyinteract with one another while wearing augmented reality (AR) smartglasses by establishing objects as personalized anchor points for socialconnection.

In sample configurations, users place physical markers on variousobjects that they use or come across in their daily lives. Using acompanion mobile device application, the user may establish connectionsbetween their physical markers and their remote partner's set ofphysical markers. Once the connections are established, electroniceyewear devices worn by the respective users will detect a physicalmarker when it is in a field of view of view of their electronic eyeweardevice. Upon recognition of a physical marker, the electronic eyeweardevice triggers visual and auditory AR content to be projected to asecond user based on the remote first user's activities and triggers theAR content to be placed at the second user's corresponding physicalmarker location (marker-endpoint). Alternatively, a setting may bechanged to have AR content placed anywhere in the direct vicinity of theremote partner (user-endpoint). Thus, the system described herein hastwo system configurations. In the case of object-to-object, content isplaced at the location of the corresponding marker. However, in theobject-to-user mode, content is loaded at a location in the vicinity ofthe receiving partner.

The duration of time in which the marker-endpoint is in the field ofview of the user's electronic eyewear device may determine what contentis placed for the remote partner. A short period of time may trigger theplacement of a simple abstract effect, such as a sparkle-like effect, atthe remote partner's marker-endpoint or user-endpoint, whereas a longerpredetermined period of time may trigger the electronic eyewear deviceto clone content from the wearer's real-world surroundings and to recordaudio for a short duration of time (i.e., 5 seconds). The cloned contentand audio is then provided to the remote partner's marker-endpoint oruser-endpoint.

The systems and methods described herein include at least the followingmain features that will become apparent from the following description:

Objects as triggers: The system enables each user to tag their own setof objects using physical markers. The user may flag specific objects asa proxy for activity detection by their electronic eyewear devices.Thereafter, when interacting with these tagged objects, the wearer'selectronic eyewear devices are triggered to spawn AR or real-worldcontent to be “placed” for their remote partner.

Objects or users as endpoints: The system also enables AR content to be“placed” in a remote partner's world using one of two strategies. Thecontent is either placed by an object tagged with the correspondingphysical marker (marker-endpoint), or the content is spawned in thevicinity of the remote partner (user-endpoint).

Real-world as a source for AR content: The system further includes aclone feature that enables visual content from the real-world to becaptured and spawned for the remote partner to experience. Also, anaudio capture feature may record ambient sounds or voices at thelocation of the marker-endpoint object, and the captured audio also maybe played back to the partner. Thus, friends may see and hear highlypersonalized and authentic content pertaining to their remote friend'sexperiences.

The following detailed description includes systems, methods,techniques, instruction sequences, and computer program productsillustrative of examples set forth in the disclosure. Numerous detailsand examples are included for the purpose of providing a thoroughunderstanding of the disclosed subject matter and its relevantteachings. Those skilled in the relevant art, however, may understandhow to apply the relevant teachings without such details. Aspects of thedisclosed subject matter are not limited to the specific devices,systems, and methods described because the relevant teachings can beapplied or practiced in a variety of ways. The terminology andnomenclature used herein is for the purpose of describing particularaspects only and is not intended to be limiting. In general, well-knowninstruction instances, protocols, structures, and techniques are notnecessarily shown in detail.

The term “connect,” “connected,” “couple,” and “coupled” as used hereinrefers to any logical, optical, physical, or electrical connection,including a link or the like by which the electrical or magnetic signalsproduced or supplied by one system element are imparted to anothercoupled or connected system element. Unless described otherwise,coupled, or connected elements or devices are not necessarily directlyconnected to one another and may be separated by intermediatecomponents, elements, or communication media, one or more of which maymodify, manipulate, or carry the electrical signals. The term “on” meansdirectly supported by an element or indirectly supported by the elementthrough another element integrated into or supported by the element.

Additional objects, advantages and novel features of the examples willbe set forth in part in the following description, and in part willbecome apparent to those skilled in the art upon examination of thefollowing and the accompanying drawings or may be learned by productionor operation of the examples. The objects and advantages of the presentsubject matter may be realized and attained by means of themethodologies, instrumentalities and combinations particularly pointedout in the appended claims.

The orientations of the electronic eyewear device, associated componentsand any complete devices incorporating an eye scanner and camera such asshown in any of the drawings, are given by way of example only, forillustration and discussion purposes. In operation for a particularvariable optical processing application, the electronic eyewear devicemay be oriented in any other direction suitable to the particularapplication of the electronic eyewear device, for example up, down,sideways, or any other orientation. Also, to the extent used herein, anydirectional term, such as front, rear, inwards, outwards, towards, left,right, lateral, longitudinal, up, down, upper, lower, top, bottom andside, are used by way of example only, and are not limiting as todirection or orientation of any optic or component of an opticconstructed as otherwise described herein.

Reference now is made in detail to the examples illustrated in theaccompanying drawings and discussed below. A sample electronic eyeweardevice and associated system for providing social connections betweenusers of electronic eyewear devices will be described with respect toFIGS. 1-16 .

The system described herein includes three main hardware components: anelectronic eyewear device, a mobile device, and a server. The electroniceyewear device will be described with respect to FIGS. 1-3 , the mobiledevice will be described with respect to FIG. 5 , and the server will bedescribed with respect to FIG. 6 . The corresponding system will bedescribed with respect to FIG. 4 . Operation of the software components,including application software on the electronic eyewear device andmobile device, as well as examples of system operation, will bedescribed with respect to FIGS. 7-16 . Such software components includesystem software for placing markers (e.g., marker-endpoints), mobiledevice software for establishing and managing the object connections,and electronic eyewear device software for recognizing the markers(e.g., objects in a scene) and for sending and receiving content.However, it will be appreciated that the mobile device and/or the servermay be removed from the system provided the electronic eyewear device isadapted to include sufficient processing and storage capabilities toperform the described functions of the mobile device and/or the server.

Electronic Eyewear Device

In sample configurations, electronic eyewear devices with augmentedreality (AR) capability are used in the systems described herein.Electronic eyewear devices are desirable to use in the system describedherein as such devices are scalable, customizable to enable personalizedexperiences, enable effects to be applied anytime, anywhere, and ensureuser privacy by enabling only the wearer to see the transmittedinformation. An electronic eyewear device such as SPECTACLES® availablefrom Snap, Inc. of Santa Monica, Calif., may be used without anyspecialized hardware in a sample configuration.

FIG. 1A illustrates a side view of an example hardware configuration ofan electronic eyewear device 100 including a right optical assembly 180Bwith an image display 180D (FIG. 2A). Electronic eyewear device 100includes multiple visible light cameras 114A-B (FIG. 3 ) that form astereo camera, of which the right visible light camera 114B is locatedon a right temple 110B and the left visible light camera 114A is locatedon a left temple 110A.

The left and right visible light cameras 114A-B may include an imagesensor that is sensitive to the visible light range wavelength. Each ofthe visible light cameras 114A-B has a different frontward facing angleof coverage, for example, visible light camera 114B has the depictedangle of coverage 111B. The angle of coverage is an angle range in whichthe image sensor of the visible light camera 114A-B picks upelectromagnetic radiation and generates images. Examples of such visiblelights camera 114A-B include a high-resolution complementarymetal-oxide-semiconductor (CMOS) image sensor and a video graphic array(VGA) camera, such as 640p (e.g., 640×480 pixels for a total of 0.3megapixels), 720p, or 1080p. Image sensor data from the visible lightcameras 114A-B may be captured along with geolocation data, digitized byan image processor, and stored in a memory.

To provide stereoscopic vision, visible light cameras 114A-B may becoupled to an image processor (element 412 of FIG. 4 ) for digitalprocessing along with a timestamp in which the image of the scene iscaptured. Image processor 412 may include circuitry to receive signalsfrom the visible light camera 114A-B and to process those signals fromthe visible light cameras 114A-B into a format suitable for storage inthe memory (element 434 of FIG. 4 ). The timestamp may be added by theimage processor 412 or other processor that controls operation of thevisible light cameras 114A-B. Visible light cameras 114A-B allow thestereo camera to simulate human binocular vision. Stereo cameras alsoprovide the ability to reproduce three-dimensional images (image 315 ofFIG. 3 ) based on two captured images (elements 358A-B of FIG. 3 ) fromthe visible light cameras 114A-B, respectively, having the sametimestamp. Such three-dimensional images 315 allow for an immersivelife-like experience, e.g., for virtual reality or video gaming. Forstereoscopic vision, the pair of images 358A-B may be generated at agiven moment in time—one image for each of the left and right visiblelight cameras 114A-B. When the pair of generated images 358A-B from thefrontward facing field of view (FOV) 111A-B of the left and rightvisible light cameras 114A-B are stitched together (e.g., by the imageprocessor 412), depth perception is provided by the optical assembly180A-B.

In an example, the electronic eyewear device 100 includes a frame 105, aright rim 107B, a right temple 110B extending from a right lateral side170B of the frame 105, and a see-through image display 180D (FIGS. 2A-B)comprising optical assembly 180B to present a graphical user interfaceto a user. The electronic eyewear device 100 includes the left visiblelight camera 114A connected to the frame 105 or the left temple 110A tocapture a first image of the scene. Electronic eyewear device 100further includes the right visible light camera 114B connected to theframe 105 or the right temple 110B to capture (e.g., simultaneously withthe left visible light camera 114A) a second image of the scene whichpartially overlaps the first image. Although not shown in FIGS. 1A-B, aprocessor 432 (FIG. 4 ) is coupled to the electronic eyewear device 100and connected to the visible light cameras 114A-B and memory 434 (FIG. 4) accessible to the processor 432, and programming in the memory 434 maybe provided in the electronic eyewear device 100 itself.

Although not shown in FIG. 1A, the electronic eyewear device 100 alsomay include a head movement tracker (element 109 of FIG. 1B) or an eyemovement tracker (element 113 of FIG. 2A or element 213 of FIGS. 2B-C).Electronic eyewear device 100 may further include the see-through imagedisplays 180C-D of optical assembly 180A-B, respectfully, for presentinga sequence of displayed images, and an image display driver (element 442of FIG. 4 ) coupled to the see-through image displays 180C-D of opticalassembly 180A-B to control the image displays 180C-D of optical assembly180A-B to present the sequence of displayed images 315, which aredescribed in further detail below. Electronic eyewear device 100 mayfurther include the memory 434 and the processor 432 having access tothe image display driver 442 and the memory 434, as well as programmingin the memory 434. Execution of the programming by the processor 432configures the electronic eyewear device 100 to perform functions,including functions to present, via the see-through image displays180C-D, an initial displayed image of the sequence of displayed images,the initial displayed image having an initial field of viewcorresponding to an initial head direction or an initial eye gazedirection as determined by the eye movement tracker 113 or 213.

Execution of the programming by the processor 432 may further configurethe electronic eyewear device 100 to detect movement of a user of theelectronic eyewear device 100 by: (i) tracking, via the head movementtracker (element 109 of FIG. 1B), a head movement of a head of the user,or (ii) tracking, via an eye movement tracker (element 113 of FIG. 2A orelement 213 of FIGS. 2B-C), an eye movement of an eye of the user of theelectronic eyewear device 100. Execution of the programming by theprocessor 432 may further configure the electronic eyewear device 100 todetermine a field of view adjustment to the initial field of view of theinitial displayed image based on the detected movement of the user. Thefield of view adjustment may include a successive field of viewcorresponding to a successive head direction or a successive eyedirection. Execution of the programming by the processor 432 may furtherconfigure the electronic eyewear device 100 to generate a successivedisplayed image of the sequence of displayed images based on the fieldof view adjustment. Execution of the programming by the processor 432may further configure the electronic eyewear device 100 to present, viathe see-through image displays 180C-D of the optical assembly 180A-B,the successive displayed images.

FIG. 1B illustrates a top cross-sectional view of the temple of theelectronic eyewear device 100 of FIG. 1A depicting the right visiblelight camera 114B, a head movement tracker 109, and a circuit board 140.Construction and placement of the left visible light camera 114A issubstantially similar to the right visible light camera 114B, except theconnections and coupling are on the left lateral side 170A (FIG. 2A). Asshown, the electronic eyewear device 100 includes the right visiblelight camera 114B and a circuit board, which may be a flexible printedcircuit board (PCB) 140. The right hinge 126B connects the right temple110B to hinged arm 125B of the electronic eyewear device 100. In someexamples, components of the right visible light camera 114B, theflexible PCB 140, or other electrical connectors or contacts may belocated on the right temple 110B or the right hinge 126B.

As shown, electronic eyewear device 100 may include a head movementtracker 109, which includes, for example, an inertial measurement unit(IMU). An inertial measurement unit is an electronic device thatmeasures and reports a body's specific force, angular rate, andsometimes the magnetic field surrounding the body, using a combinationof accelerometers and gyroscopes, sometimes also magnetometers. Theinertial measurement unit works by detecting linear acceleration usingone or more accelerometers and rotational rate using one or moregyroscopes. Typical configurations of inertial measurement units containone accelerometer, gyro, and magnetometer per axis for each of the threeaxes: horizontal axis for left-right movement (X), vertical axis (Y) fortop-bottom movement, and depth or distance axis for up-down movement(Z). The accelerometer detects the gravity vector. The magnetometerdefines the rotation in the magnetic field (e.g., facing south, north,etc.) like a compass that generates a heading reference. The threeaccelerometers detect acceleration along the horizontal, vertical, anddepth axis defined above, which can be defined relative to the ground,the electronic eyewear device 100, or the user wearing the electroniceyewear device 100.

Electronic eyewear device 100 may detect movement of the user of theelectronic eyewear device 100 by tracking, via the head movement tracker109, the head movement of the head of the user. The head movementincludes a variation of head direction on a horizontal axis, a verticalaxis, or a combination thereof from the initial head direction duringpresentation of the initial displayed image on the image display. In oneexample, tracking, via the head movement tracker 109, the head movementof the head of the user includes measuring, via the inertial measurementunit 109, the initial head direction on the horizontal axis (e.g., Xaxis), the vertical axis (e.g., Y axis), or the combination thereof(e.g., transverse or diagonal movement). Tracking, via the head movementtracker 109, the head movement of the head of the user further includesmeasuring, via the inertial measurement unit 109, a successive headdirection on the horizontal axis, the vertical axis, or the combinationthereof during presentation of the initial displayed image.

Tracking, via the head movement tracker 109, the head movement of thehead of the user may further include determining the variation of headdirection based on both the initial head direction and the successivehead direction. Detecting movement of the user of the electronic eyeweardevice 100 may further include in response to tracking, via the headmovement tracker 109, the head movement of the head of the user,determining that the variation of head direction exceeds a deviationangle threshold on the horizontal axis, the vertical axis, or thecombination thereof. In sample configurations, the deviation anglethreshold is between about 3° to 10°. As used herein, the term “about”when referring to an angle means ±10% from the stated amount.

Variation along the horizontal axis slides three-dimensional objects,such as characters, Bitmojis, application icons, etc. in and out of thefield of view by, for example, hiding, unhiding, or otherwise adjustingvisibility of the three-dimensional object. Variation along the verticalaxis, for example, when the user looks upwards, in one example, displaysweather information, time of day, date, calendar appointments, etc. Inanother example, when the user looks downwards on the vertical axis, theelectronic eyewear device 100 may power down.

As shown in FIG. 1B, the right temple 110B includes temple body 211 anda temple cap, with the temple cap omitted in the cross-section of FIG.1B. Disposed inside the right temple 110B are various interconnectedcircuit boards, such as PCBs or flexible PCBs 140, that includecontroller circuits for right visible light camera 114B, microphone(s)130, speaker(s) 132, low-power wireless circuitry (e.g., for wirelessshort-range network communication via BLUETOOTH®), and high-speedwireless circuitry (e.g., for wireless local area network communicationvia WI-FI®).

The right visible light camera 114B is coupled to or disposed on theflexible PCB 140 and covered by a visible light camera cover lens, whichis aimed through opening(s) formed in the right temple 110B. In someexamples, the frame 105 connected to the right temple 110B includes theopening(s) for the visible light camera cover lens. The frame 105 mayinclude a front-facing side configured to face outwards away from theeye of the user. The opening for the visible light camera cover lens maybe formed on and through the front-facing side. In the example, theright visible light camera 114B has an outward facing angle of coverage111B with a line of sight or perspective of the right eye of the user ofthe electronic eyewear device 100. The visible light camera cover lensalso can be adhered to an outward facing surface of the right temple110B in which an opening is formed with an outwards facing angle ofcoverage, but in a different outwards direction. The coupling can alsobe indirect via intervening components.

Left (first) visible light camera 114A may be connected to the leftsee-through image display 180C of left optical assembly 180A to generatea first background scene of a first successive displayed image. Theright (second) visible light camera 114B may be connected to the rightsee-through image display 180D of right optical assembly 180B togenerate a second background scene of a second successive displayedimage. The first background scene and the second background scene maypartially overlap to present a three-dimensional observable area of thesuccessive displayed image.

Flexible PCB 140 may be disposed inside the right temple 110B andcoupled to one or more other components housed in the right temple 110B.Although shown as being formed on the circuit boards 140 of the righttemple 110B, the right visible light camera 114B can be formed on thecircuit boards 140 of the left temple 110A, the hinged arms 125A-B, orframe 105.

FIG. 2A illustrates a rear view of an example hardware configuration ofan electronic eyewear device 100. As shown in FIG. 2A, the electroniceyewear device 100 is in a form configured for wearing by a user, whichare eyeglasses in the example of FIG. 2A. The electronic eyewear device100 can take other forms and may incorporate other types of frameworks,for example, a headgear, a headset, or a helmet.

In the eyeglasses example, electronic eyewear device 100 includes theframe 105 which includes the left rim 107A connected to the right rim107B via the bridge 106 adapted for a nose of the user. The left andright rims 107A-B include respective apertures 175A-B which hold therespective optical element 180A-B, such as a lens and the see-throughdisplays 180C-D. As used herein, the term lens is meant to covertransparent or translucent pieces of glass or plastic having curved andflat surfaces that cause light to converge/diverge or that cause littleor no convergence/divergence.

Although shown as having two optical elements 180A-B, the electroniceyewear device 100 can include other arrangements, such as a singleoptical element depending on the application or intended user of theelectronic eyewear device 100. As further shown, electronic eyeweardevice 100 includes the left temple 110A adjacent the left lateral side170A of the frame 105 and the right temple 110B adjacent the rightlateral side 170B of the frame 105. The temples 110A-B may be integratedinto the frame 105 on the respective sides 170A-B (as illustrated) orimplemented as separate components attached to the frame 105 on therespective sides 170A-B. Alternatively, the temples 110A-B may beintegrated into hinged arms 125A-B attached to the frame 105.

In the example of FIG. 2A, an eye scanner 113 may be provided thatincludes an infrared emitter 115 and an infrared camera 120. Visiblelight cameras typically include a blue light filter to block infraredlight detection. In an example, the infrared camera 120 is a visiblelight camera, such as a low-resolution video graphic array (VGA) camera(e.g., 640×480 pixels for a total of 0.3 megapixels), with the bluefilter removed. The infrared emitter 115 and the infrared camera 120 maybe co-located on the frame 105. For example, both are shown as connectedto the upper portion of the left rim 107A. The frame 105 or one or moreof the left and right temples 110A-B may include a circuit board (notshown) that includes the infrared emitter 115 and the infrared camera120. The infrared emitter 115 and the infrared camera 120 can beconnected to the circuit board by soldering, for example.

Other arrangements of the infrared emitter 115 and infrared camera 120may be implemented, including arrangements in which the infrared emitter115 and infrared camera 120 are both on the right rim 107B, or indifferent locations on the frame 105. For example, the infrared emitter115 may be on the left rim 107A and the infrared camera 120 may be onthe right rim 107B. In another example, the infrared emitter 115 may beon the frame 105 and the infrared camera 120 may be on one of thetemples 110A-B, or vice versa. The infrared emitter 115 can be connectedessentially anywhere on the frame 105, left temple 110A, or right temple110B to emit a pattern of infrared light. Similarly, the infrared camera120 can be connected essentially anywhere on the frame 105, left temple110A, or right temple 110B to capture at least one reflection variationin the emitted pattern of infrared light.

The infrared emitter 115 and infrared camera 120 may be arranged to faceinwards towards an eye of the user with a partial or full field of viewof the eye in order to identify the respective eye position and gazedirection. For example, the infrared emitter 115 and infrared camera 120may be positioned directly in front of the eye, in the upper part of theframe 105 or in the temples 110A-B at either ends of the frame 105.

FIG. 2B illustrates a rear view of an example hardware configuration ofanother electronic eyewear device 200. In this example configuration,the electronic eyewear device 200 is depicted as including an eyescanner 213 on a right temple 210B. As shown, an infrared emitter 215and an infrared camera 220 are co-located on the right temple 210B. Itshould be understood that the eye scanner 213 or one or more componentsof the eye scanner 213 can be located on the left temple 210A and otherlocations of the electronic eyewear device 200, for example, the frame105. The infrared emitter 215 and infrared camera 220 are like that ofFIG. 2A, but the eye scanner 213 can be varied to be sensitive todifferent light wavelengths as described previously in FIG. 2A. Similarto FIG. 2A, the electronic eyewear device 200 includes a frame 105 whichincludes a left rim 107A which is connected to a right rim 107B via abridge 106. The left and right rims 107A-B may include respectiveapertures which hold the respective optical elements 180A-B comprisingthe see-through display 180C-D.

FIGS. 2C-D illustrate rear views of example hardware configurations ofthe electronic eyewear device 100, including two different types ofsee-through image displays 180C-D. In one example, these see-throughimage displays 180C-D of optical assembly 180A-B include an integratedimage display. As shown in FIG. 2C, the optical assemblies 180A-Binclude a suitable display matrix 180C-D of any suitable type, such as aliquid crystal display (LCD), an organic light-emitting diode (OLED)display, a waveguide display, or any other such display.

The optical assembly 180A-B also includes an optical layer or layers176, which can include lenses, optical coatings, prisms, mirrors,waveguides, optical strips, and other optical components in anycombination. The optical layers 176A-N can include a prism having asuitable size and configuration and including a first surface forreceiving light from display matrix and a second surface for emittinglight to the eye of the user. The prism of the optical layers 176A-N mayextend over all or at least a portion of the respective apertures 175A-Bformed in the left and right rims 107A-B to permit the user to see thesecond surface of the prism when the eye of the user is viewing throughthe corresponding left and right rims 107A-B. The first surface of theprism of the optical layers 176A-N faces upwardly from the frame 105 andthe display matrix overlies the prism so that photons and light emittedby the display matrix impinge the first surface. The prism may be sizedand shaped so that the light is refracted within the prism and isdirected towards the eye of the user by the second surface of the prismof the optical layers 176A-N. In this regard, the second surface of theprism of the optical layers 176A-N can be convex to direct the lighttowards the center of the eye. The prism can optionally be sized andshaped to magnify the image projected by the see-through image displays180C-D, and the light travels through the prism so that the image viewedfrom the second surface is larger in one or more dimensions than theimage emitted from the see-through image displays 180C-D.

In another example, the see-through image displays 180C-D of opticalassembly 180A-B may include a projection image display as shown in FIG.2D. The optical assembly 180A-B includes a projector 150, which may be athree-color projector using a scanning mirror, a galvanometer, a laserprojector, or other types of projectors. During operation, an opticalsource such as a projector 150 is disposed in or on one of the temples110A-B of the electronic eyewear device 100. Optical assembly 180A-B mayinclude one or more optical strips 155A-N spaced apart across the widthof the lens of the optical assembly 180A-B or across a depth of the lensbetween the front surface and the rear surface of the lens.

As the photons projected by the projector 150 travel across the lens ofthe optical assembly 180A-B, the photons encounter the optical strips155A-N. When a particular photon encounters a particular optical strip,the photon is either redirected towards the user's eye, or it passes tothe next optical strip. A combination of modulation of projector 150,and modulation of optical strips, may control specific photons or beamsof light. In an example, a processor controls optical strips 155A-N byinitiating mechanical, acoustic, or electromagnetic signals. Althoughshown as having two optical assemblies 180A-B, the electronic eyeweardevice 100 can include other arrangements, such as a single or threeoptical assemblies, or the optical assembly 180A-B may have arrangeddifferent arrangement depending on the application or intended user ofthe electronic eyewear device 100.

As further shown in FIGS. 2C-D, electronic eyewear device 100 includes aleft temple 110A adjacent the left lateral side 170A of the frame 105and a right temple 110B adjacent the right lateral side 170B of theframe 105. The temples 110A-B may be integrated into the frame 105 onthe respective lateral sides 170A-B (as illustrated) or implemented asseparate components attached to the frame 105 on the respective sides170A-B. Alternatively, the temples 110A-B may be integrated into thehinged arms 125A-B attached to the frame 105.

In one example, the see-through image displays include the firstsee-through image display 180C and the second see-through image display180D. Electronic eyewear device 100 may include first and secondapertures 175A-B that hold the respective first and second opticalassembly 180A-B. The first optical assembly 180A may include the firstsee-through image display 180C (e.g., a display matrix 177 of FIG. 2C oroptical strips and a projector (not shown) in left temple 110A). Thesecond optical assembly 180B may include the second see-through imagedisplay 180D (e.g., a display matrix of FIG. 2C or optical strips 155A-Nand a projector 150 in right temple 110B). The successive field of viewof the successive displayed image may include an angle of view betweenabout 15° to 30, and more specifically 24°, measured horizontally,vertically, or diagonally. The successive displayed image having thesuccessive field of view represents a combined three-dimensionalobservable area visible through stitching together of two displayedimages presented on the first and second image displays.

As used herein, “an angle of view” describes the angular extent of thefield of view associated with the displayed images presented on each ofthe left and right image displays 180C-D of optical assembly 180A-B. The“angle of coverage” describes the angle range that a lens of visiblelight cameras 114A-B or infrared camera 220 can image. Typically, theimage circle produced by a lens is large enough to cover the film orsensor completely, possibly including some vignetting (i.e., a reductionof an image's brightness or saturation toward the periphery compared tothe image center). If the angle of coverage of the lens does not fillthe sensor, the image circle will be visible, typically with strongvignetting toward the edge, and the effective angle of view will belimited to the angle of coverage. The “field of view” is intended todescribe the field of observable area which the user of the electroniceyewear device 100 can see through his or her eyes via the displayedimages presented on the left and right image displays 180C-D of theoptical assembly 180A-B. Image display 180C of optical assembly 180A-Bcan have a field of view with an angle of coverage between 15° to 30°,for example 24°, and have a resolution of 480×480 pixels.

FIG. 3 illustrates an example of capturing visible light with cameras114A-B. Visible light is captured by the left visible light camera 114Awith a round field of view (FOV). 111A. A chosen rectangular left rawimage 358A is used for image processing by image processor 412 (FIG. 4). Visible light is also captured by the right visible light camera 114Bwith a round FOV 111B. A rectangular right raw image 358B chosen by theimage processor 412 is used for image processing by processor 412. Basedon processing of the left raw image 358A and the right raw image 358Bhaving an overlapping field of view 313, a three-dimensional image 315of a three-dimensional scene, referred to hereafter as an immersiveimage, is generated by processor 412 and displayed by displays 180C and180D and which is viewable by the user.

FIG. 4 illustrates a high-level functional block diagram includingexample electronic components disposed in electronic eyewear device 100or 200. The illustrated electronic components include the processor 432,the memory 434, and the see-through image display 180C and 180D.

Memory 434 includes instructions for execution by processor 432 toimplement the functionality of electronic eyewear devices 100 and 200,including instructions for processor 432 to control in the image 315.Such functionality may be implemented by processing instructions of eyetracking software 445, object/marker recognition and connection software460, and image segmentation software 470 that is stored in memory 434and executed by high speed processor 432. Processor 432 receives powerfrom battery 450 and executes the instructions stored in memory 434, orintegrated with the processor 432 on-chip, to perform the functionalityof electronic eyewear devices 100 and 200 and to communicate withexternal devices via wireless connections.

The electronic eyewear devices 100 and 200 may incorporate an eyemovement tracker 445 (e.g., shown as infrared emitter 215 and infraredcamera 220 in FIG. 2B) and may provide user interface adjustments via amobile device 500 (FIG. 5 ) and a server system 498 connected viavarious networks. Mobile device 500 may be a smartphone, tablet, laptopcomputer, access point, or any other such device capable of connectingwith the electronic eyewear devices 100 or 200 using both a low-powerwireless connection 425 and a high-speed wireless connection 437. Mobiledevice 500 is further connected to server system 498 via a network 495.The network 495 may include any combination of wired and wirelessconnections.

Electronic eyewear devices 100 and 200 may include at least two visiblelight cameras 114A-B (one associated with the left lateral side 170A andone associated with the right lateral side 170B). Electronic eyeweardevices 100 and 200 further include two see-through image displays180C-D of the optical assembly 180A-B (one associated with the leftlateral side 170A and one associated with the right lateral side 170B).Electronic eyewear devices 100 and 200 also include image display driver442, image processor 412, low-power circuitry 420, and high-speedcircuitry 430. The components shown in FIG. 4 for the electronic eyeweardevices 100 and 200 are located on one or more circuit boards, forexample, a PCB or flexible PCB 140, in the temples. Alternatively, oradditionally, the depicted components can be located in the temples,frames, hinges, hinged arms, or bridge of the electronic eyewear devices100 and 200. Left and right visible light cameras 114A-B can includedigital camera elements such as a complementarymetal-oxide-semiconductor (CMOS) image sensor, charge coupled device, alens, or any other respective visible or light capturing elements thatmay be used to capture data, including images of scenes with unknownobjects.

Eye movement tracking programming 445 implements the user interfacefield of view adjustment instructions, including instructions to causethe electronic eyewear devices 100 or 200 to track, via the eye movementtracker 213, the eye movement of the eye of the user of the electroniceyewear devices 100 or 200. Other implemented instructions (functions)cause the electronic eyewear devices 100 and 200 to determine the FOVadjustment to the initial FOV 111A-B based on the detected eye movementof the user corresponding to a successive eye direction. Furtherimplemented instructions generate a successive displayed image of thesequence of displayed images based on the field of view adjustment. Thesuccessive displayed image is produced as visible output to the user viathe user interface. This visible output appears on the see-through imagedisplays 180C-D of optical assembly 180A-B, which is driven by imagedisplay driver 442 to present the sequence of displayed images,including the initial displayed image with the initial field of view andthe successive displayed image with the successive field of view.

The object/marker recognition and connection programming 460 and imagesegmentation programming 470 will be described in further detail belowwith respect to FIG. 16 .

As shown in FIG. 4 , high-speed circuitry 430 includes high-speedprocessor 432, memory 434, and high-speed wireless circuitry 436. In theexample, the image display driver 442 is coupled to the high-speedcircuitry 430 and operated by the high-speed processor 432 in order todrive the left and right image displays 180C-D of the optical assembly180A-B. High-speed processor 432 may be any processor capable ofmanaging high-speed communications and operation of any generalcomputing system needed for electronic eyewear device 100 or 200.High-speed processor 432 includes processing resources needed formanaging high-speed data transfers on high-speed wireless connection 437to a wireless local area network (WLAN) using high-speed wirelesscircuitry 436. In certain examples, the high-speed processor 432executes an operating system such as a LINUX operating system or othersuch operating system of the electronic eyewear device 100 or 200 andthe operating system is stored in memory 434 for execution. In additionto any other responsibilities, the high-speed processor 432 executing asoftware architecture for the electronic eyewear device 100 or 200 isused to manage data transfers with high-speed wireless circuitry 436. Incertain examples, high-speed wireless circuitry 436 is configured toimplement Institute of Electrical and Electronic Engineers (IEEE) 802.11communication standards, also referred to herein as WI-FI®. In otherexamples, other high-speed communications standards may be implementedby high-speed wireless circuitry 436.

Low-power wireless circuitry 424 and the high-speed wireless circuitry436 of the electronic eyewear devices 100 and 200 can include shortrange transceivers (BLUETOOTH®) and wireless wide, local, or wide areanetwork transceivers (e.g., cellular or WI-FI®). Mobile device 500,including the transceivers communicating via the low-power wirelessconnection 425 and high-speed wireless connection 437, may beimplemented using details of the architecture of the electronic eyeweardevice 100 and 200, as can other elements of network 495.

Memory 434 includes any storage device capable of storing various dataand applications, including, among other things, color maps, camera datagenerated by the left and right visible light cameras 114A-B and theimage processor 412, as well as images generated for display by theimage display driver 442 on the see-through image displays 180C-D of theoptical assembly 180A-B. While memory 434 is shown as integrated withhigh-speed circuitry 430, in other examples, memory 434 may be anindependent standalone element of the electronic eyewear device 100 or200. In certain such examples, electrical routing lines may provide aconnection through a system on chip that includes the high-speedprocessor 432 from the image processor 412 or low-power processor 422 tothe memory 434. In other examples, the high-speed processor 432 maymanage addressing of memory 434 such that the low-power processor 422will boot the high-speed processor 432 any time that a read or writeoperation involving memory 434 is needed.

Server system 498 may be one or more computing devices as part of aservice or network computing system, for example, that includes aprocessor, a memory, and network communication interface to communicateover the network 495 with the mobile device 500 and electronic eyeweardevices 100 and 200. Electronic eyewear devices 100 and 200 may beconnected with a host computer. For example, the electronic eyeweardevices 100 or 200 may be paired with the mobile device 500 via thehigh-speed wireless connection 437 or connected to the server system 498via the network 495. Also, as explained in more detail below, a gallery480 of snapshots and AR objects may be maintained by the backend serversystem 498 for each user and invoked by communications providing linksto the stored snapshots and AR objects.

Output components of the electronic eyewear devices 100 and 200 includevisual components, such as the left and right image displays 180C-D ofoptical assembly 180A-B as described in FIGS. 2C-D (e.g., a display suchas a liquid crystal display (LCD), a plasma display panel (PDP), a lightemitting diode (LED) display, a projector, or a waveguide). The imagedisplays 180C-D of the optical assembly 180A-B are driven by the imagedisplay driver 442. The output components of the electronic eyeweardevices 100 and 200 further include acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor), other signalgenerators, and so forth. The input components of the electronic eyeweardevices 100 and 200, the mobile device 500, and server system 498, mayinclude alphanumeric input components (e.g., a keyboard, a touch screenconfigured to receive alphanumeric input, a photo-optical keyboard, orother alphanumeric input components), point-based input components(e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, orother pointing instruments), tactile input components (e.g., a physicalbutton, a touch screen that provides location and force of touches ortouch gestures, or other tactile input components), audio inputcomponents (e.g., a microphone), and the like.

Electronic eyewear devices 100 and 200 may optionally include additionalperipheral device elements such as ambient light and spectral sensors,biometric sensors, heat sensor 440, or other display elements integratedwith electronic eyewear device 100 or 200. For example, the peripheraldevice elements may include any I/O components including outputcomponents, motion components, position components, or any other suchelements described herein. The electronic eyewear devices 100 and 200can take other forms and may incorporate other types of frameworks, forexample, a headgear, a headset, or a helmet.

For example, the biometric components of the electronic eyewear devices100 and 200 may include components to detect expressions (e.g., handexpressions, facial expressions, vocal expressions, body gestures, oreye tracking), measure biosignals (e.g., blood pressure, heart rate,body temperature, perspiration, or brain waves), identify a person(e.g., voice identification, retinal identification, facialidentification, fingerprint identification, or electroencephalogrambased identification), and the like. The motion components includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The position components include location sensor components to generatelocation coordinates (e.g., a Global Positioning System (GPS) receivercomponent), WI-FI® or BLUETOOTH® transceivers to generate positioningsystem coordinates, altitude sensor components (e.g., altimeters orbarometers that detect air pressure from which altitude may be derived),orientation sensor components (e.g., magnetometers), and the like. Suchpositioning system coordinates can also be received over wirelessconnections 425 and 437 from the mobile device 500 via the low-powerwireless circuitry 424 or high-speed wireless circuitry 436.

Mobile Device

FIG. 5 illustrates a sample configuration of a mobile device 500 adaptedto manage social connections via objects in sample configurations. Inparticular, FIG. 5 is a high-level functional block diagram of anexample mobile device 500 that a user may use to manage socialconnections via objects as described herein. Mobile device 500 mayinclude a flash memory 505 that stores programming to be executed by theCPU 510 to perform all or a subset of the functions described herein.For example, the flash memory may store object pairing and connectionmanagement software 515 for execution by CPU 510 to enable the user ofthe mobile device 500 to establish objects as markers and to manageconnections as described herein with respect to FIG. 15 . The mobiledevice 500 may further include a camera 525 that comprises one or morevisible-light cameras (first and second visible-light cameras withoverlapping fields of view) or at least one visible-light camera and adepth sensor with substantially overlapping fields of view. Flash memory505 may further include multiple images or video, which are generatedvia the camera 525.

The mobile device 500 may further include an image display 530, a mobiledisplay driver 535 to control the image display 530, and a displaycontroller 540. In the example of FIG. 5 , the image display 530 mayinclude a user input layer 545 (e.g., a touchscreen) that is layered ontop of or otherwise integrated into the screen used by the image display530. Examples of touchscreen-type mobile devices that may be usedinclude (but are not limited to) a smart phone, a personal digitalassistant (PDA), a tablet computer, a laptop computer, or other portabledevice. However, the structure and operation of the touchscreen-typedevices is provided by way of example; the subject technology asdescribed herein is not intended to be limited thereto. For purposes ofthis discussion, FIG. 5 therefore provides a block diagram illustrationof the example mobile device 500 with a user interface that includes atouchscreen input layer 545 for receiving input (by touch, multi-touch,or gesture, and the like, by hand, stylus, or other tool) and an imagedisplay 530 for displaying content.

As shown in FIG. 5 , the mobile device 500 includes at least one digitaltransceiver (XCVR) 550, shown as WWAN XCVRs, for digital wirelesscommunications via a wide-area wireless mobile communication network.The mobile device 500 also may include additional digital or analogtransceivers, such as short-range transceivers (XCVRs) 555 forshort-range network communication, such as via NFC, VLC, DECT, ZigBee,Bluetooth™, or WI-FI®. For example, short range XCVRs 555 may take theform of any available two-way wireless local area network (WLAN)transceiver of a type that is compatible with one or more standardprotocols of communication implemented in wireless local area networks,such as one of the WI-FI® standards under IEEE 802.11.

To generate location coordinates for positioning of the mobile device500, the mobile device 500 also may include a global positioning system(GPS) receiver. Alternatively, or additionally, the mobile device 500may utilize either or both the short range XCVRs 555 and WWAN XCVRs 550for generating location coordinates for positioning. For example,cellular network, WI-FI®, or Bluetooth™ based positioning systems maygenerate very accurate location coordinates, particularly when used incombination. Such location coordinates may be transmitted to the mobiledevice 500 over one or more network connections via XCVRs 550, 555.

The transceivers 550, 555 (i.e., the network communication interface)may conform to one or more of the various digital wireless communicationstandards utilized by modern mobile networks. Examples of WWANtransceivers 550 include (but are not limited to) transceiversconfigured to operate in accordance with Code Division Multiple Access(CDMA) and 3rd Generation Partnership Project (3GPP) networktechnologies including, for example and without limitation, 3GPP type 2(or 3GPP2) and LTE, at times referred to as “4G.” The transceivers mayalso incorporate broadband cellular network technologies referred to as“5G.” For example, the transceivers 550, 555 provide two-way wirelesscommunication of information including digitized audio signals, stillimage and video signals, web page information for display as well asweb-related inputs, and various types of mobile message communicationsto/from the mobile device 500.

The mobile device 500 may further include a microprocessor thatfunctions as the central processing unit (CPU) 510. A processor is acircuit having elements structured and arranged to perform one or moreprocessing functions, typically various data processing functions.Although discrete logic components could be used, the examples utilizecomponents forming a programmable CPU. A microprocessor for exampleincludes one or more integrated circuit (IC) chips incorporating theelectronic elements to perform the functions of the CPU 510. The CPU510, for example, may be based on any known or available microprocessorarchitecture, such as a Reduced Instruction Set Computing (RISC) usingan ARM architecture, as commonly used today in mobile devices and otherportable electronic devices. Of course, other arrangements of processorcircuitry may be used to form the CPU 510 or processor hardware insmartphone, laptop computer, and tablet.

The CPU 510 serves as a programmable host controller for the mobiledevice 500 by configuring the mobile device 500 to perform variousoperations, for example, in accordance with instructions or programmingexecutable by CPU 510. For example, such operations may include variousgeneral operations of the mobile device 500, as well as operationsrelated to the programming for messaging apps and AR camera applicationson the mobile device 500. Although a processor may be configured by useof hardwired logic, typical processors in mobile devices are generalprocessing circuits configured by execution of programming.

The mobile device 500 further includes a memory or storage system, forstoring programming and data. In the example shown in FIG. 5 , thememory system may include flash memory 505, a random-access memory (RAM)560, and other memory components 565, as needed. The RAM 560 may serveas short-term storage for instructions and data being handled by the CPU510, e.g., as a working data processing memory. The flash memory 505typically provides longer-term storage.

Hence, in the example of mobile device 500, the flash memory 505 may beused to store programming or instructions for execution by the CPU 510.Depending on the type of device, the mobile device 500 stores and runs amobile operating system through which specific applications areexecuted. Examples of mobile operating systems include Google Android,Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS,RIM BlackBerry OS, or the like.

Finally, the mobile device 500 may include an audio transceiver 570 thatmay receive audio signals from the environment via a microphone (notshown) and provide audio output via a speaker (not shown). Audio signalsmay be coupled with video signals and other messages by a messagingapplication or social media application implemented on the mobile device500.

Backend Server System

Techniques described herein also may be used with one or more of thecomputer systems described herein or with one or more other systems. Forexample, the various procedures described herein may be implemented withhardware or software, or a combination of both. For example, at leastone of the processor, memory, storage, output device(s), inputdevice(s), or communication connections discussed below can each be atleast a portion of one or more hardware components. Dedicated hardwarelogic components can be constructed to implement at least a portion ofone or more of the techniques described herein. For example, and withoutlimitation, such hardware logic components may includeField-programmable Gate Arrays (FPGAs), Program-specific IntegratedCircuits (ASICs), Program-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), etc. Applications that may include the apparatus and systems ofvarious aspects can broadly include a variety of electronic and computersystems. Techniques may be implemented using two or more specificinterconnected hardware modules or devices with related control and datasignals that can be communicated between and through the modules, or asportions of an application-specific integrated circuit. Additionally,the techniques described herein may be implemented by software programsexecutable by a computer system. As an example, implementations caninclude distributed processing, component/object distributed processing,and parallel processing. Moreover, virtual computer system processingcan be constructed to implement one or more of the techniques orfunctionality, as described herein.

FIG. 6 illustrates a sample configuration of a computer system adaptedto implement the systems and methods described herein. In particular,FIG. 6 illustrates a block diagram of an example of a machine 600 uponwhich one or more configurations of the backend server system 498 (FIG.4 ) may be implemented. As described herein, the backend server system498 may execute instructions for connecting the IDs, images, anddescriptions of respective marker-endpoint objects or user-endpointobjects and for storing communications of AR generated objects (e.g.,sparkles) and/or user-generated objects (e.g., snapshot of coffee mug)received from a first user for transmission to a second user uponreceipt of an indication that the second user is a user-endpoint or hasviewed the user's corresponding marker-endpoint with the user'selectronic eyewear device. The backend server system 498 may alsomaintain a gallery 480 of user snapshots and AR objects. In alternativeconfigurations, the machine 600 may operate as a standalone device ormay be connected (e.g., networked) to other machines. In a networkeddeployment, the machine 600 may operate in the capacity of a servermachine, a client machine, or both in server-client networkenvironments. In an example, the machine 600 may act as a peer machinein peer-to-peer (P2P) (or other distributed) network environment. Insample configurations, the machine 600 may be a personal computer (PC),a tablet PC, a set-top box (STB), a personal digital assistant (PDA), amobile telephone, a smart phone, a web appliance, a server, a networkrouter, switch or bridge, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. For example, machine 600 may serve as a workstation, afront-end server, or a back-end server of a communication system.Machine 600 may implement the methods described herein by running thesoftware used to implement the features for controlling IoT devices asdescribed herein. Further, while only a single machine 600 isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein, such as cloud computing, software as aservice (SaaS), other computer cluster configurations.

Examples, as described herein, may include, or may operate on,processors, logic, or a number of components, modules, or mechanisms(herein “modules”). Modules are tangible entities (e.g., hardware)capable of performing specified operations and may be configured orarranged in a certain manner. In an example, circuits may be arranged(e.g., internally or with respect to external entities such as othercircuits) in a specified manner as a module. In an example, the whole orpart of one or more computer systems (e.g., a standalone, client orserver computer system) or one or more hardware processors may beconfigured by firmware or software (e.g., instructions, an applicationportion, or an application) as a module that operates to performspecified operations. In an example, the software may reside on amachine readable medium. The software, when executed by the underlyinghardware of the module, causes the hardware to perform the specifiedoperations.

Accordingly, the term “module” is understood to encompass at least oneof a tangible hardware or software entity, be that an entity that isphysically constructed, specifically configured (e.g., hardwired), ortemporarily (e.g., transitorily) configured (e.g., programmed) tooperate in a specified manner or to perform part or all of any operationdescribed herein. Considering examples in which modules are temporarilyconfigured, each of the modules need not be instantiated at any onemoment in time. For example, where the modules comprise ageneral-purpose hardware processor configured using software, thegeneral-purpose hardware processor may be configured as respectivedifferent modules at different times. Software may accordingly configurea hardware processor, for example, to constitute a particular module atone instance of time and to constitute a different module at a differentinstance of time.

Machine (e.g., computer system) 600 may include a hardware processor 602(e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 604 and a static memory 606, some or all of which may communicatewith each other via an interlink (e.g., bus) 608. The machine 600 mayfurther include a display unit 610 (shown as a video display), analphanumeric input device 612 (e.g., a keyboard), and a user interface(UI) navigation device 614 (e.g., a mouse). In an example, the displayunit 610, input device 612 and UI navigation device 614 may be a touchscreen display. The machine 600 may additionally include a mass storagedevice (e.g., drive unit) 616, a signal generation device 618 (e.g., aspeaker), a network interface device 620, and one or more sensors 622.Example sensors 622 include one or more of a global positioning system(GPS) sensor, compass, accelerometer, temperature, light, camera, videocamera, sensors of physical states or positions, pressure sensors,fingerprint sensors, retina scanners, or other sensors. The machine 600also may include an output controller 624, such as a serial (e.g.,universal serial bus (USB), parallel, or other wired or wireless (e.g.,infrared(IR), near field communication (NFC), etc.) connection tocommunicate or control one or more peripheral devices (e.g., a printer,card reader, etc.).

The mass storage device 616 may include a machine readable medium 626 onwhich is stored one or more sets of data structures or instructions 628(e.g., software) embodying or utilized by any one or more of thetechniques or functions described herein. The instructions 628 may alsoreside, completely or at least partially, within the main memory 604,within static memory 606, or within the hardware processor 602 duringexecution thereof by the machine 600. In an example, one or anycombination of the hardware processor 602, the main memory 604, thestatic memory 606, or the mass storage device 616 may constitute machinereadable media.

While the machine readable medium 626 is illustrated as a single medium,the term “machine readable medium” may include a single medium ormultiple media (e.g., at least one of a centralized or distributeddatabase, or associated caches and servers) configured to store the oneor more instructions 628. The term “machine readable medium” may includeany medium that is capable of storing, encoding, or carryinginstructions for execution by the machine 600 and that cause the machine600 to perform any one or more of the techniques of the presentdisclosure, or that is capable of storing, encoding, or carrying datastructures used by or associated with such instructions. Non-limitingmachine readable medium examples may include solid-state memories, andoptical and magnetic media. Specific examples of machine readable mediamay include non-volatile memory, such as semiconductor memory devices(e.g., Electrically Programmable Read-Only Memory (EPROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM)) and flash memorydevices; magnetic disks, such as internal hard disks and removabledisks; magneto-optical disks; Random Access Memory (RAM); Solid StateDrives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machinereadable media may include non-transitory machine-readable media. Insome examples, machine readable media may include machine readable mediathat is not a transitory propagating signal.

The instructions 628 may further be transmitted or received overcommunications network 632 using a transmission medium via the networkinterface device 620. The machine 600 may communicate with one or moreother machines utilizing any one of a number of transfer protocols(e.g., frame relay, internet protocol (IP), transmission controlprotocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). Example communication networks may include alocal area network (LAN), a wide area network (WAN), a packet datanetwork (e.g., the Internet), mobile telephone networks (e.g., cellularnetworks), Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., Institute of Electrical and Electronics Engineers (IEEE)802.11 family of standards known as WI-FI®), IEEE 802.15.4 family ofstandards, a Long Term Evolution (LTE) family of standards, a UniversalMobile Telecommunications System (UMTS) family of standards,peer-to-peer (P2P) networks, among others. In an example, the networkinterface device 620 may include one or more physical jacks (e.g.,Ethernet, coaxial, or phone jacks) or one or more antennas 630 toconnect to the communications network 632. In an example, the networkinterface device 620 may include a plurality of antennas 630 towirelessly communicate using at least one of single-inputmultiple-output (SIMO), multiple-input multiple-output (MIMO), ormultiple-input single-output (MISO) techniques. In some examples, thenetwork interface device 620 may wirelessly communicate using MultipleUser MIMO techniques.

The features and flow charts described herein can be embodied in on oneor more methods as method steps or in one more applications as describedpreviously. According to some configurations, an “application” or“applications” are program(s) that execute functions defined in theprograms. Various programming languages can be employed to generate oneor more of the applications, structured in a variety of manners, such asobject-oriented programming languages (e.g., Objective-C, Java, or C++)or procedural programming languages (e.g., C or assembly language). In aspecific example, a third party application (e.g., an applicationdeveloped using the ANDROID™ or IOS™ software development kit (SDK) byan entity other than the vendor of the particular platform) may bemobile software running on a mobile operating system such as IOS™,ANDROID™ WINDOWS® Phone, or another mobile operating systems. In thisexample, the third party application can invoke API calls provided bythe operating system to facilitate functionality described herein. Theapplications can be stored in any type of computer readable medium orcomputer storage device and be executed by one or more general purposecomputers. In addition, the methods and processes disclosed herein canalternatively be embodied in specialized computer hardware or anapplication specific integrated circuit (ASIC), field programmable gatearray (FPGA) or a complex programmable logic device (CPLD).

Program aspects of the technology may be thought of as “products” or“articles of manufacture” typically in the form of at least one ofexecutable code or associated data that is carried on or embodied in atype of machine readable medium. For example, programming code couldinclude code for the touch sensor or other functions described herein.“Storage” type media include any or all of the tangible memory of thecomputers, processors or the like, or associated modules thereof, suchas various semiconductor memories, tape drives, disk drives and thelike, which may provide non-transitory storage at any time for thesoftware programming. All or portions of the software may at times becommunicated through the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another. Thus, another typeof media that may bear the programming, media content or meta-data filesincludes optical, electrical, and electromagnetic waves, such as usedacross physical interfaces between local devices, through wired andoptical landline networks and over various air-links. The physicalelements that carry such waves, such as wired or wireless links, opticallinks, or the like, also may be considered as media bearing thesoftware. As used herein, unless restricted to “non-transitory”,“tangible”, or “storage” media, terms such as computer or machine“readable medium” refer to any medium that participates in providinginstructions or data to a processor for execution.

Hence, a machine readable medium may take many forms of tangible storagemedium. Non-volatile storage media include, for example, optical ormagnetic disks, such as any of the storage devices in any computer(s) orthe like, such as may be used to implement the client device, mediagateway, transcoder, etc. shown in the drawings. Volatile storage mediainclude dynamic memory, such as main memory of such a computer platform.Tangible transmission media include coaxial cables, copper wire andfiber optics, including the wires that comprise a bus within a computersystem. Carrier-wave transmission media may take the form of electric orelectromagnetic signals, or acoustic or light waves such as thosegenerated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer may read at least one of programmingcode or data. Many of these forms of computer readable media may beinvolved in carrying one or more sequences of one or more instructionsto a processor for execution.

Distributed and Connected Real-World Objects

To implement the functionality for providing social connections betweenobjects as described herein, two software applications are implementedon the hardware described above. One application runs on the mobiledevice 500 (iPhone/Android) and one runs on the electronic eyeweardevice 100. Both partners in a pair use both applications to implementthe functionality.

In a sample configuration, the mobile device application 515 (FIG. 15 )is installed on a mobile device 500 by each user by, for example,scanning a Snap Code available from Snap, Inc. of Santa Monica, Calif.Each user logs into the mobile device application 515 with their logininformation. Once the user is signed in and has identified theirmetadata (i.e., a Pair ID and a user assignment, User A or User B), theuser can place markers and take photos of their locations to be storedin the mobile device application. Once a pair of corresponding markershas been set up by each user, respectively, a connection is establishedbetween them through which AR exchanges can occur. The respective users'electronic eyewear devices 100 are paired to the respective mobiledevice applications to leverage this connection information.

On the other hand, the electronic eyewear device application 460 (FIG.16 ) is installed on the user's electronic eyewear device 100 and allowseach user to experience (see, listen) to content from the remote friend.An electronic eyewear device 100 including the electronic eyewear deviceapplication 460 will be able to detect the user's physical markers andwill load auditory and visual content from the remote partner for theuser to experience. In the examples described herein, two forms ofvisual AR content may be sent: 1) AR content denoting user presence,such as floating sparkles, and 2) cloned visual content, selectedsnapshots or AR content with or without a recorded audio snippet whichis extracted from the real-world environment.

The system so configured enables remote friends to indirectly interactwith one another while wearing augmented reality (AR) electronic eyeweardevices 100 by establishing objects as personalized anchor points forsocial connection. The system allows friends to be aware of each other'sstate—what they are doing as they interact and use different objectsthroughout the day—by leaving traces, whether indoors or outdoors. Theusers place physical markers on various objects that they use or comeacross in their daily lives. The physical markers are a proxy to actualobject detection and may be generated dynamically. Using the companionmobile device application 515, the user may establish connectionsbetween their physical markers and their remote partner's set ofphysical markers. The users may set symmetric connections (such as fromlamp-to-lamp) or asymmetric connections (such as from lamp-to-mug). Onceset, an electronic eyewear device 100 running the electronic eyeweardevice application 460 may detect the physical marker when the physicalmarker is in the field of view of the electronic eyewear device 100,thereby triggering AR content to be projected for the user (visual andauditory content), based on the remote partner's activities and ARcontent to be placed at the remote partner's corresponding markerlocation (marker-endpoint).

In sample configurations, the duration of time in which the marker is inthe wearer's field of view determines what content is placed for theremote partner. A time-buffer is used to track the duration of time inwhich the marker is in the field of view. A short period of timetriggers the placement of predetermined AR content, such as an abstractsparkle-like effect, at the remote partner's marker-endpoint, while alonger period of time triggers the electronic eyewear device to clonecontent from the wearer's real-world surroundings or to select prestoredAR content, as well as to record audio for a short duration of time(i.e., 5 seconds). The system also may support transient and persistentAR. Depending on the setting, the AR content on the receiver side canaccumulate and build up (persistent) or fade away after viewing(transient). As time passes, the AR content's color and/or brightnessmay fade away as well to indicate how long ago an activity was performedby the user.

The systems and methods described herein thus allow users to interactwith and share their state with friends by just looking at objectshaving pre-set physical markers around them. The physical markers may befixed objects in the user's surroundings but may also be faces, pets,vehicles, and the other movable objects such as a person. The systemsand methods provide passive, hands-free messaging by simply looking at(scanning) a particular object or marker to indicate their state. Usingthe system, a user can send messages from one object to another, or froman object to the user directly, anywhere the user may be located. Once aseries of objects with marker-endpoints are set up, a user can simplywalk in their home or outdoors, go through their routine, and just bylooking at the marker-endpoints, their friends will be notified abouttheir activities and actions. In sample configurations, looking at amarker-endpoint object triggers and sends a default AR content (e.g.,sparkles) to the receiver at a specific location or anywhere to whichthe marker-endpoint object has been connected. Conversely, while lookingat a marker-endpoint object, the system can recommend relevant ARcontent, for utilitarian or expressive purposes, that the user may useto send to their friend. For example, a user of the electronic eyeweardevice 100 may scan a scene and recommend AR content including, forexample, a set of AR Lenses of the type available from Snap, Inc. ofSanta Monica, Calif., that the user can select and send to a friend.

In the case where a face is used as a marker, messages may be triggeredwhen the user looks at the face. For example, if a user selects SuniLee's face as a marker, every time the user watches her perform, theuser's friend is notified. The friend would thus be informed that theuser is watching gymnastics right now. Similarly, if a user selectstheir friend's face, any time they see their friend, a message will betriggered indicating that they both are together.

The systems and methods described herein further allow users to create3D snapshots of real-world objects to indicate their state and place(realistic) AR content as if the actual object was at their friend'slocation. This can also lead to the creation of a gallery, public orprivate, and serve as a marketplace. In this case, the system enables auser to create a clone of a real-world object and share it with theirfriend to indicate their state or context—as if that object was in theirfriend's space. For example, the user may provide a passive snapshot. Ifthe user selects a mug as a marker, then every time the user looks at orscans the mug, a snapshot is created and sent to the connected friend asa realistic AR mug. Each snapshot can indicate the type and the level ofcoffee remaining in the mug by synchronizing the state between the realmug and an AR mug. On the other hand, in the case of an active snapshot,a user may scan an object such as a flower while taking a walk and placethe flower at a friend's desk, remotely in AR, to indicate that the useris taking a walk. Similarly, the user may send a snapshot of a new dressto indicate that she is shopping.

Just like picture, gif and video galleries available today, users maycreate a gallery 480 (FIG. 4 ) of 3D snapshots, private or public, thatusers can use to indicate their state or mood. This approach can createa significant repository of realistic AR content that is invoked byproviding links in communications to enable access by users via theirelectronic eyewear devices 100 and a marketplace for objects that can bebought/rented/leased. In addition to individual users, businesses cancreate snapshots of the food or artifacts that a user can access via amap. In this case, users may scan AR objects that show up on the mapwhere the objects were scanned. Restaurants can scan the food before itis sent and people can see the food that is to be delivered to them.Users may scan all different pieces of a place to create a virtual placeon the map that represents the real-life one. In a sample configuration,the snapshot is a 3D model of the real world object that is so realisticthat the receiver feels as if the sender has sent the object itself. Inthis example, the 3D model may blend seamlessly with the physicalenvironment as if it were a real object.

In other applications, users may place something from their world intotheir friend's world, keep a copy of that object forever, give multiplecopies of the object to different people, modify or augment the digitalobject, put that digital object into their own augmented reality(AR)/virtual reality (VR) world, use the object as a building block fora bigger object, and/or simulate presence by discussing an object as ifthat object was right between two different users.

Operation of the systems and methods will become apparent from thefollowing illustrative operational examples.

It is assumed that a pair of close friends (user 1 and user 2) who livea significant distance from each other would like to stay sociallyconnected to each other using their electronic eyewear devices 100. Todo so, user 1 establishes a local object as a marker-endpoint. To do so,user 1 identifies an object 700 (e.g., refrigerator) in her apartment(FIG. 7 ) by selecting an image of the object 700 as a marker-endpointat 710. The mobile device application 515 of user 1's mobile device 500₁ then provides an object identifier and a picture of the selectedobject 700 to the backend server system 498. The object identifier mayinclude a name provided by user 1. User 1 then uses the mobile deviceapplication 515 to connect the identified object 700 to an objectsimilarly identified by user 2. The IDs, pictures, and provided namesfor the objects marked by the respective users are stored in anapplication of the backend server system 498 as part of a social mediacommunications platform connecting user 1 and user 2. As noted above,the object markers are a proxy to actual object detection and may begenerated dynamically. A plurality of such connections may beestablished between user 1 and user 2. The connections may be symmetric(such as from refrigerator-to-refrigerator) or asymmetric (such as fromlamp-to-mug) and may be 1-1, 1-N, N-1, or N-N connections, where N is aninteger.

As illustrated in FIG. 8 , user 2 similarly identifies an object 800(e.g., a cabinet) in his apartment by selecting an image of the object800 as a marker and connecting the object 800 to one or more of user 1'sobjects (e.g., refrigerator 700) using a mobile device application ofuser 2's mobile device 500 ₂ at 810. Providing a name such as“refrigerator” may facilitate such connections.

Now that a connection has been made between user 1's refrigerator 700and user 2's cabinet 800, the system is ready to implement the socialconnectivity functionality. To activate the social connectivityfeatures, user 1 simply glances at her refrigerator 700 while wearingher electronic eyewear device 100 ₁. As shown in FIG. 9 , the electroniceyewear device 100 ₁ scans user 1's surroundings and identifies therefrigerator 700 using object recognition capabilities of the electroniceyewear device 100 ₁. To scan the user's surroundings formarker-endpoint objects or to identify objects to establish asmarker-endpoint objects, the user's electronic eyewear device 100 maymonitor the user's gaze direction and linear or rotational movement ofthe user's head to track the scene. A visual scan by the electroniceyewear device 100 can be activated with scan initiation means such as abutton tap or a press and hold of a scan button at any time. In the caseof a visual scan, the captured image may be forwarded to a trainedneural network of a deep learning model on the electronic eyewear device100 or to backend services available on the backend server system 498accessible to the electronic eyewear device 100 to process the capturedimage to identify objects in the scene. A voice scan, on the other hand,may be initiated by a “wake word,” which is a phrase that wakes theelectronic eyewear device 100 from sleep to trigger a scan by the camera150 or to trigger a search for objects by voice-to-text processing ofthe user's voice to extract keywords that are matched to objects in thescene. In either case, “signal descriptor text” may be presented to adisplay of the electronic eyewear device 100 as objects in the capturedscene or words in the captured voice are recognized. The objects havinga recognition score above a set confidence score may be identified aspotential marker objects to be provided to the display of the electroniceyewear device 100 in response to the scan. Scan notifications such assounds or displayed words or icons may be used to indicate when abackground scan has been initiated. When the background scan has beencompleted, a notification of the completed scan results may be providedto the display.

Upon recognizing the refrigerator 700 in the scanned image, theelectronic eyewear device 100 ₁ initiates a transmission of a simplecommunication to user 2 indicating that user 1 is active and has viewedthe refrigerator 700. For example, the electronic eyewear device 100 ₁may initiate the transmission of a communication that includes a link toinvoke a preselected AR image or animation such as sparkles from user 1to user 2 by simply glancing at the refrigerator 700 established as themarker—endpoint between user 1 and user 2. The electronic eyewear device100 ₁ may optionally present to user 1's display a representation of awormhole 900 that is activated when the refrigerator 700 is viewed byuser 1 and presents an animation showing the sparkles being sucked intothe wormhole 900 for transmission via the wormhole 900 to user 2. Theanimation may also include corresponding sound effects. Optionally, user1's interaction with refrigerator 700 may be timestamped and thetimestamp information provided with the communication (e.g., with thesparkles).

To receive the communication (sparkles) from user 1, user 2 simply putson his electronic eyewear device 100 ₂ and glances at hismarker-endpoint object 800 (e.g., cabinet) connected to user 1's object700 (e.g., refrigerator). Upon user 2's electronic eyewear device 100 ₂recognition of the object 800, any communication associated with object800 is pushed from the backend server system 498 to user 2's electroniceyewear device 100 ₂. As shown in FIG. 10 , upon recognition of theobject 800, the sparkles 1000 invoked by the communication from user 1are received and displayed as an overlay on the display of user 2'selectronic eyewear device 100 ₂. The electronic eyewear device 100 ₂ mayoptionally present to user 2's display a representation of a wormhole1010 that is activated and presents an animation showing the sparklesbeing shot out of the wormhole 1010 to the display of user 2'selectronic eyewear device 100 ₂. The animation may also includecorresponding sound effects. Such presentation of the sparkles 1000indicates to user 2 that user 1 is awake and active and has glanced ather refrigerator 700.

As another example, it is assumed that user 2 would like to respond touser 1 by sending a communication showing what he is doing and that heis thinking of user 1. User 2 decides to show user 1 that he is drinkingcoffee from a mug that user 1 gave to user 2 as a gift. As noted above,the duration of time in which user 2's marker (e.g., cabinet 800) is inuser 2's field of view may determine what content is placed for user 1.A short period of time may trigger the placement of a simple abstracteffect, such as the sparkle effect received from user 1. However, arecognition by user 2's electronic eyewear device 100 ₂ that user 2 hasbeen viewing the cabinet 800 for a longer predetermined period of timemay trigger user 2's electronic eyewear device 100 ₂ to clone contentfrom user 2's real-world surroundings. In this case, user 2 may elect tocapture a snapshot of mug 1100 that user 1 gave to user 2 as a gift. Themug 1100 may be extracted from the captured snapshot using imagesegmentation software 470 of user 2's electronic eyewear device 100 ₂.Alternatively, the segmented image may be processed by image processingsoftware of the backend server system 498 to provide a 2D or a 3Drendering of the segmented image. User 2's electronic eyewear device 100₂ may also present user 2 with the option of recording audio for a shortduration of time (i.e., 5 seconds) to send with the segmented image ofmug 1100.

Once the segmented mug image 1100 and the audio recording are captured,user 2 may swipe forward or provide a recognized gesture to transmit aghost image of the mug 1100 with the audio recording to user 1. As shownin FIG. 11 , the electronic eyewear device 100 ₂ may optionally presentto the display of user 2's electronic eyewear device 100 ₂ arepresentation of the ghost image of the mug 1100 in front of wormhole1010. User 2's electronic eyewear device 100 ₂ may also present ananimation showing the wormhole 1010 being activated and sucking a ghostimage of the mug 1100 into the wormhole 1010, along with associatedsound effects.

To receive the communication including the image of mug 1100 from user2, user 1 simply glances at her marker-endpoint object 700 (e.g.,refrigerator) connected to user 2's object 800 (e.g., cabinet). Uponuser 1's electronic eyewear device 100 ₁ recognition of the object 700,any communication associated with object 700 is pushed from the backendserver system 498 to user 1's electronic eyewear device 100 ₁. As shownin FIG. 12 , upon recognition of the object 700, the snapshot of mug1100 from user 2 is received and displayed as an overlay on the displayof user 1's electronic eyewear device 100 ₁. The electronic eyeweardevice 100 ₁ may optionally present to user 1's display a representationof a wormhole 900 that is activated and presents an animation showingthe image of the mug 1100 appearing out of the wormhole 900 to thedisplay of user 1's electronic eyewear device 100 ₁, along with optionalsound effects associated with the presentation of the image. Suchpresentation of the image of the mug 1100 along with the playback of theaudio recording from user 2 indicates to user 1 that user 2 is drinkingcoffee from a mug that user 1 gave to user 2 as a gift. Thus, user 1 mayappreciate that user 2 thought of user 1 during user 2's coffee break.

As illustrated in FIGS. 13 and 14 , the connections of marker-endpointobjects between user 1's apartment 1300 and user 2's apartment 1310 forsocially connecting user 1 and user 2 may be symmetric (between likeobjects) or asymmetric (between different types of objects) and may bebetween stationary and moving objects. Also, as noted above, theconnections may be 1-1, 1-N, N-1, and N-N, so any combination ofmarker-endpoint objects may be used to connect the environments 1300 and1310 of user 1 and user 2. The connected objects may provide a socialgraph connecting respective objects within the respective surroundingsof the respective users. Of course, the users have to make more effortto keep track of the respective ends of the wormholes when asymmetricconnections are used.

In FIG. 13 , sparkles 1320 are sent from user 1 to user 2 (andvice-versa) to indicate that the respective users are present.Similarly, in FIG. 14 , the snapshot of mug 1100 and the snapshot ofcereal bowl 1400 are sent between user 1 and user 2 as indicated tofurther the social connectedness of user 1 and user 2 by indicating thatwhat each is doing at the time the transmission was sent. Of course, thesnapshots or segmented images could be any object, including an image ofanother friend or group of friends.

FIG. 15 illustrates a flow chart for functionality implemented by themobile device 500 associated with the electronic eyewear device 100 toprovide object pairing and to manage connections in a sampleconfiguration. Such functionality may be implemented as objectpairing/connection management software 515 in FIG. 5 .

As indicated in FIG. 15 , the mobile device 500 receives anidentification tag (ID) at 1500 for a marker-endpoint object selected bythe user of the associated electronic eyewear device 100. The user ispresented with the option at 1510 to provide a name for the identifiedmarker object. For example, if the marker object is the user'srefrigerator, the user may name the marker “refrigerator.” Such namingfacilitates pairing by another user. In certain configurations, it maybe desirable to name another user as the marker object, in which case,any AR objects or images of real-world objects would proceed directly tothe mobile device 500 and/or the electronic mobile device of the otheruser. In this latter case, the other user would be a user-endpointobject.

At 1520, the marker object ID, its picture, and its name are stored onthe backend server system 498 to facilitate pairing with other users. At1530, the user may access the marker objects of another user for pairingwith the identified marker object. In this case, the other user's markerobjects with their pictures, names, and IDs are presented to the displayof the user's mobile device 500 for selection. The selected markerobjects of the other user are paired with the user's marker object andthe connection is stored in the backend server system 498. The otheruser's marker object may also include the user herself In this case, theID would be the user ID and the image would be an image of the user. Theuser endpoint would be the IP address of the user's mobile device 500 orelectronic eyewear device 100. The user may also manage her connectionsby updating the marker objects of the other user that are connected tomarker objects in the surroundings of the user. Once the connectionshave been so established, the AR content stored in the gallery 480 ofthe backend server system 498 and/or a snapshot of the object providedby the electronic eyewear device 100 may be invoked by communicationsto/from the backend server system 498 and the other user's electroniceyewear device 100 ₂ at 1540 when the respective users scan or glance attheir respective marker objects that are the respective marker-endpointobjects of the connection(s) between the users.

FIG. 16 illustrates a flow chart for functionality implemented by theelectronic eyewear device 100 to identify and recognize marker objectsand to prompt the sending of AR objects and/or real-world images fromthe environment for socially connecting respective users in a sampleconfiguration. Such functionality may be implemented as object/markerrecognition and connection software 460 in FIG. 4 .

As illustrated in FIG. 16 , the electronic eyewear device 100 isinitiated at 1600 by initiating a voice scan or a visual scan asdescribed above. The scanned scene may be forwarded to a trained neuralnetwork of a deep learning model on the electronic eyewear device 100 orto backend services available on the backend server system 498accessible to the electronic eyewear device 100 to process the capturedimage to identify objects in the scene. An ID for the identifiedobject(s) is provided for tracking purposes, and the ID and picture ofthe object may be provided to the associated mobile device 500 forpairing (see FIG. 15 ) at 1610. However, such pairing is not necessaryas the message may be sent using any conventional communication device,such as SMS text or email.

If the identified object is already paired, then the electronic eyeweardevice 100 may receive and display at 1620 any content that has beenprovided by another user having an object paired to the identifiedobject. This content may be stored at the backend server system 498 andinvoked by a communication to the user's electronic eyewear device 100upon detection of the paired object in the scanned image. As notedabove, depending on the setting, the AR content may accumulate and buildup (persistent data) or fade away after viewing (transient data). Overtime, the AR content's color may fade away to indicate how long ago anactivity was performed by the user. The time of the activity may also berecorded, and the AR content may be stored in a message gallery 480, asdesired.

In sample configurations, the electronic eyewear device 100 furtherkeeps track of how long the user looks at the paired marker-object inorder to determine what type of message should be sent to a paired user.If it is determined at 1630 that the user has only glanced at the pairedobject for no more than a predetermined short duration (e.g., x seconds,where x is 1-5), then a preselected AR object such as sparkles isinvoked by a communication sent to the paired user at 1640. On the otherhand, if it is determined at 1630 that the user has looked at the pairedobject for at least the predetermined short duration, then the user maybe prompted at 1650 to take a snapshot (active snapshot) or to select aprestored snapshot (passive snapshot) from the environment or to selecta prestored AR object from an AR object gallery 480 to send to thepaired user. At 1660, the user option is captured. If the user selectedto send a snapshot, image segmentation software 470 (FIG. 4 ) optionallymay be activated at 1670 to segment an object from the active or passivesnapshot to send to the paired user, or the snapshot may be sent withoutmodification. On the other hand, if the user has selected a prestored ARobject from the AR object gallery 480, then the selected AR object isinvoked by a communication sent to the user. For example, rather thansending an extracted snapshot of a mug in the above example, a prestoredsnapshot of the mug or a 3D representation of the mug that was prestoredin the AR object gallery 480 may be invoked by the communication instead(e.g., by a link to the storage location in the AR object gallery 480 ofthe prestored snapshot of the mug or 3D representation of the mug). At1680, the selected snapshot or AR object or segmented object from thereal-world scene is invoked by a communication sent to the mobile device500 over the network 495 to the backend server system 498. The backendserver system 498, in turn, provides the invoked image to the electroniceyewear device 100 of the other user for viewing adjacent the pairedmarker object(s) when the other user views the paired marker object(s).When the paired marker object is the other user, the image may beprovided directly to the electronic eyewear device 100 of the other userfor display in the vicinity of the other user. Alternatively, theelectronic eyewear device 100 may communicate directly with the backendserver system 498, provided the electronic eyewear device 100 has therequisite circuitry to communicate directly over an Internet connection.However, the communication may be picked up by the second user byconventional means without pairing, such as directly via theirelectronic eyewear device 100 or their mobile device 500.

Users may share information with each other in many different scenariosusing the system and methods described herein. For example, user 1 mayestablish a marker-endpoint object as her mirror and send images of whatclothing she has elected to wear for the day. User 2 may place amarker-endpoint object near his piano and send sparkles and an audioclip each time he sits down to play his piano. The users may also shareimages of their meals while they eat.

It will be further appreciated that the backend server system 498 maymaintain a gallery 480 of AR content and images that users haveexchanged with each other via particular connections much in the sameway that SMS messaging systems maintain a record of texts sent back andforth between users or messaging systems such as SNAPCHAT® availablefrom Snap, Inc. of Santa Monica, Calif., maintain communications in aMemories feature. The stored AR content and images may be presented tothe display of the user's electronic eyewear device 100 for selection,as desired, in the event that the user wishes to resend a previouslysent image. In particular configurations, the AR content may be Lensesof the type available from Snap, Inc. of Santa Monica, Calif.

Separately, it will be further appreciated that companies may providelogos on their products that may be used as respective marker-endpointobjects to establish communications “wormholes” with the logos of otherusers or the support personnel for the product. Such communicationsnetworks may be used to facilitate social connectedness through scannedbased messaging amongst users of a product.

In another alternative configuration, rather than simply gazing at amarker-endpoint object, the electronic eyewear device 100 may track theglobal positioning system (GPS) coordinates of an object duringrespective scans in a session. Then, when the object is moved betweenscans in a session, the communication of AR elements (e.g., sparkles) orscanned objects may be triggered. Similarly, the endpoint object may bethe paired user's mobile device, whereby the AR object or segmentedimage is provided to the paired user's paired electronic eyewear device100 irrespective of the paired user's location.

In yet another alternative configuration, the marker-endpoint objectsmay be connected to endpoints in a map to provide a snapshot of 2D or 3Dimages from one or more users to a portion of a map that is being viewedby another user. For example, an image of a cheesesteak may be presentedto the viewer's electronic eyewear device 100 when the viewer viewsPhiladelphia on a map.

It will be appreciated by those skilled in the art that the methodsdescribed herein may be initiated and conducted without any particulargestures or touch operations. The actions may be activated from passiveprocessing of the images in the scene to trigger the indicated effectswhen the marker-endpoint object is viewed, for example. The imageextraction may be conducted by staring at the marker-endpoint object forthe predetermined duration of time and then focusing on the object to beextracted and sent, all without any hand gestures or manual selection.

The scope of protection is limited solely by the claims that now follow.That scope is intended and should be interpreted to be as broad as isconsistent with the ordinary meaning of the language that is used in theclaims when interpreted in light of this specification and theprosecution history that follows and to encompass all structural andfunctional equivalents. Notwithstanding, none of the claims are intendedto embrace subject matter that fails to satisfy the requirement ofSections 101, 102, or 103 of the Patent Act, nor should they beinterpreted in such a way. Any unintended embracement of such subjectmatter is hereby disclaimed.

Except as stated immediately above, nothing that has been stated orillustrated is intended or should be interpreted to cause a dedicationof any component, step, feature, object, benefit, advantage, orequivalent to the public, regardless of whether it is or is not recitedin the claims.

It will be understood that the terms and expressions used herein havethe ordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”“includes,” “including,” or any other variation thereof, are intended tocover a non-exclusive inclusion, such that a process, method, article,or apparatus that comprises or includes a list of elements or steps doesnot include only those elements or steps but may include other elementsor steps not expressly listed or inherent to such process, method,article, or apparatus. An element preceded by “a” or “an” does not,without further constraints, preclude the existence of additionalidentical elements in the process, method, article, or apparatus thatcomprises the element.

Unless otherwise stated, any and all measurements, values, ratings,positions, magnitudes, sizes, and other specifications that are setforth in this specification, including in the claims that follow, areapproximate, not exact. Such amounts are intended to have a reasonablerange that is consistent with the functions to which they relate andwith what is customary in the art to which they pertain. For example,unless expressly stated otherwise, a parameter value or the like mayvary by as much as ±10% from the stated amount.

In addition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in various examples for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimed examplesrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, the subject matter to be protected liesin less than all features of any single disclosed example. Thus, thefollowing claims are hereby incorporated into the Detailed Description,with each claim standing on its own as a separately claimed subjectmatter.

While the foregoing has described what are considered to be the bestmode and other examples, it is understood that various modifications maybe made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that they may be appliedin numerous applications, only some of which have been described herein.It is intended by the following claims to claim any and allmodifications and variations that fall within the true scope of thepresent concepts.

What is claimed is:
 1. An electronic eyewear device adapted to be wornon the head of a first user, comprising: at least one camera arranged tocapture an image in an environment of the first user; a memory thatstores instructions; and a processor that executes the instructions toperform operations including: scanning a scene using the at least onecamera to capture at least one image in the environment of the firstuser; identifying at least one physical marker in the scanned scene; andupon identification of the at least one physical marker in the scannedscene, passively sending a message to at least one of a directly to asecond user or to at least one physical marker at a remote location forpresentation to a second user, wherein the message is sent without useof the first user's hands to make a selection.
 2. The electronic eyeweardevice of claim 1, wherein the at least one physical marker in thescanned scene is associated with at least one of a fixed object, a humanface, a pet, a vehicle, or a person.
 3. The electronic eyewear device ofclaim 1, wherein execution of the instructions cause the processor toperform additional operations including: passively sending messages tothe at least one physical marker at the remote location for presentationto the second user as the first user moves through the environment ofthe first user and encounters physical markers.
 4. The electroniceyewear device of claim 1, wherein the message comprises preselected ARcontent.
 5. The electronic eyewear device of claim 1, wherein executionof the instructions cause the processor to perform additional operationsincluding: upon identification of the at least one physical marker inthe scanned scene, recommending AR content for the first user to send tothe second user.
 6. The electronic eyewear device of claim 2, wherein,when the at least one physical marker in the scanned scene is associatedwith a human face, execution of the instructions cause the processor toperform additional operations including: automatically sending themessage for presentation to the second user whenever the human face isin a field of view of the electronic eyewear device.
 7. The electroniceyewear device of claim 2, wherein the fixed object is a logo and thefirst user has a product with a first copy of the logo and the seconduser has a product with a second copy of the logo, and wherein when theat least one physical marker in the scanned scene is associated with afirst copy of the logo, execution of the instructions cause theprocessor to perform additional operations including: automaticallysending the message for presentation to the second user whenever thefirst copy of the logo is in a field of view of the electronic eyeweardevice.
 8. The electronic eyewear device of claim 2, wherein the atleast one physical marker at the remote location is associated with thesecond user, and wherein execution of the instructions cause theprocessor to perform additional operations including: uponidentification of the at least one physical marker in the scanned scene,automatically sending the message to a mobile device of the second userfor presentation in a vicinity of the second user.
 9. The electroniceyewear device of claim 1, wherein execution of the instructions causethe processor to perform additional operations including: trackingglobal positioning system (GPS) coordinates of an object duringrespective scans of the scene; and when the object is moved betweenscans of the scene, triggering sending of the message to the at leastone physical marker at the remote location for presentation to thesecond user.
 10. The electronic eyewear device of claim 1, whereinexecution of the instructions cause the processor to perform additionaloperations including: determining that the at least one physical markerin the scanned scene has been within a field of view of the electroniceyewear device for a predetermined amount of time; determining that anobject not associated with the at least one physical marker has beenwithin the field of view of the electronic eyewear device for anotherpredetermined amount of time; extracting the object from an imagecaptured within the field of view of the electronic eyewear device; andpassively sending the extracted object to the at least one physicalmarker at the remote location for presentation to the second user.
 11. Amethod of providing scan based imaging using an electronic eyeweardevice, including: scanning a scene using the electronic eyewear deviceto capture at least one image in an environment of a first user;identifying at least one physical marker in the scanned scene; and uponidentification of the at least one physical marker in the scanned scene,passively sending a message to at least one of directly to a second useror to at least one physical marker at a remote location for presentationto the second user, wherein the message is sent without use of the firstuser's hands to make a selection.
 12. The method of claim 11, furtherincluding: passively sending messages to the at least one physicalmarker at the remote location for presentation to the second user as thefirst user moves through the environment of the first user andencounters physical markers.
 13. The method of claim 11, wherein themessage comprises preselected AR content.
 14. The method of claim 11,further including: upon identification of the at least one physicalmarker in the scanned scene, recommending AR content for the first userto send to the second user.
 15. The method of claim 11, wherein the atleast one physical marker in the scanned scene is associated with ahuman face, the method further including: automatically sending themessage for presentation to the second user whenever the human face isin a field of view of the electronic eyewear device.
 16. The method ofclaim 11, wherein at least one physical marker in the scanned scene isassociated with a logo and the first user has a product with a firstcopy of the logo and the second user has a product with a second copy ofthe logo, and wherein when the at least one physical marker in thescanned scene is associated with a first copy of the logo, the methodfurther including: automatically sending the message for presentation tothe second user whenever the first copy of the logo is in a field ofview of the electronic eyewear device.
 17. The method of claim 11,further including: tracking global positioning system (GPS) coordinatesof an object during respective scans of the scene; and when the objectis moved between scans of the scene, triggering sending of the messageto an endpoint address of the at least one physical marker at the remotelocation for presentation to the second user.
 18. The method of claim11, wherein the at least one physical marker at the remote location isassociated with the second user, the method further including: uponidentification of the at least one physical marker in the scanned scene,automatically sending the message to a mobile device of the second userfor presentation in a vicinity of the second user.
 19. The method ofclaim 11, further including: determining that the at least one physicalmarker in the scanned scene has been within a field of view of theelectronic eyewear device for a predetermined amount of time;determining that an object not associated with the at least one physicalmarker has been within the field of view of the electronic eyeweardevice for another predetermined amount of time; extracting the objectfrom an image captured within the field of view of the electroniceyewear device; and passively sending the extracted object to the atleast one physical marker at the remote location for presentation to thesecond user.
 20. A non-transitory computer-readable storage medium thatstores instructions that when executed by at least one processor causethe at least one processor to provide scan based imaging using anelectronic eyewear device by performing operations including: scanning ascene using the electronic eyewear device to capture at least one imagein an environment of a first user; identifying at least one physicalmarker in the scanned scene; and passively sending messages to at leastone of directly to a second user or to at least one physical marker at aremote location for presentation to the second user as the first usermoves through the environment of the first user and encounters physicalmarkers, wherein the messages are sent without use of the first user'shands to make a selection.